Examining the Effectiveness of Testing Coverage Tools: An Empirical Study

Code coverage is one of the most important aspects of software testing, which helps software engineers to understand which portion of code has been executed using a given test suite throughout the software testing process. Automatic testing tools are widely utilized to provide testing coverage metrics in order to gauge the quality of software, but these tools may have some shortcomings such as the difference among the values of code coverage metric of a given program using different code coverage tools. Therefore, we designed and performed a controlled experiment to investigate whether these tools have a significant difference among the measured values of coverage metric or not. We collected the coverage data that consist of branch, line, statement, and method coverage metrics. Statistically, our findings show that there is a significant difference of code coverage results among the code coverage tools in terms of branch and method coverage metrics.